Three-dimensional computer graphics models are used in many computer graphics applications. Generating 3D models manually is time consuming, and causes a bottleneck for many practical applications. Besides the difficulty of modeling complex shapes, it is often impossible to replicate the geometry and appearance of complex objects using prior art parametric reflectance models.
Not surprisingly, systems for generating 3D models automatically by scanning or imaging physical objects have greatly increased in significance. An ideal system would acquire the shape and appearance of an object automatically, and construct a detailed 3D model that can be placed in an arbitrary realistic scene with arbitrary novel illumination.
Although there has been much recent work towards this goal, no system to date fulfills all of these requirements. Many systems, including most commercial systems, focus on capturing accurate shape, but neglect to acquire an accurate appearance. Other methods capture reflectance properties of 3D objects and fit these properties to parametric bi-directional reflectance distribution functions (BRDFs). However, those methods fail for complex anisotropic BRDFs and do not model important appearance effects, such as inter-reflections, self-shadowing, translucency, sub-surface light scattering, or refraction.
There have also been a number of image-based methods for acquiring and representing complex objects. But they either lack a 3D shape model, assume accurate 3D geometry, do not allow rendering the objects under novel arbitrary illumination, or are restricted to a single viewpoint. All of these systems require substantial manual involvement.
There are many methods for acquiring high-quality 3D shape from physical objects, including contact digitizers, passive stereo depth-extraction, and active light imaging systems. Passive digitizers are inadequate where the object being digitized does not have sufficient texture. Nearly all passive methods assume that the BRDF is Lambertian, or does not vary across the surface.
Magda et al., in “Beyond Lambert: Re-constructing Surfaces with Arbitrary BRDFs.” Proc. of IEEE International Conference on Computer Vision ICCV, 2001, described a stereopsis method that uses the Helmholtz reciprocity to extract depth maps from objects with arbitrary BRDFs. However, their method is not robust for smooth objects. In addition, their method does not take inter-reflections and self-shadowing into account.
Active light systems, such as laser range scanners, are very popular and have been employed to acquire large models in the field, see Levoy et al. “The Digital Michelangelo Project: 3D Scanning of Large Statues,” Computer Graphics, SIGGRAPH 2000 Proceedings, pp. 131-144, 2000, and Rushmeier et al. “Acquiring Input for Rendering at Appropriate Levels of Detail: Digitizing a Piet{grave over ( )}a,” Proceedings of the 9th Eurographics Workshop on Rendering, pp. 81-92, 1998.
Active light systems often require a registration step to align separately acquired scanned meshes, see Curless et al., “A Volumetric Method for Building Complex Models from Range Images,” Computer Graphics, SIGGRAPH 96 Proceedings, pp. 303-312, 1996, and Turk et al., “Zippered Polygon Meshes from Range Images,” Computer Graphics, SIGGRAPH 94 Proceedings, pp. 311-318, 1994. Alternatively, the scanned geometry is aligned with separately acquired texture images, see Bernardini et al., “High-Quality Texture Reconstruction from Multiple Scans,” IEEE Trans. on Vis. and Comp. Graph., 7(4):318-332, 2001.
Often, filling of gaps due to missing data is necessary as well. Systems have been constructed where multiple lasers are used to acquire a surface color estimate along lines-of-sight of the imaging system. However, those systems are not useful for capturing objects under realistic illumination. All active light systems place restrictions on the types of materials that can be scanned, as described in detail by Hawkins et al., in “A Photometric Approach to Digitizing Cultural Artifacts,” 2nd International Symposium on Virtual Reality, Archaeology, and Cultural Heritage, 2001.
To render objects constructed of arbitrary materials, image-based rendering can be used. Image-based representations have the advantage of capturing and representing an object regardless of the complexity of its geometry and appearance. Prior art image-based methods allowed for navigation within a scene using correspondence information, see Chen et al., “View Interpolation for Image Synthesis,” Computer Graphics,” SIGGRAPH 93 Proceedings, pp. 279-288, 1993, and McMillan et al., “Plenoptic Modeling: An Image-Based Rendering System,” Computer Graphics, SIGGRAPH 95 Proceedings, pp. 39-46, 1995. Because this method does not construct a model of the 3D object, it is severely limited.
Light field methods achieve similar results without geometric information, but with an increased number of images, see Gortler et al, “The Lumigraph,” Computer Graphics, SIGGRAPH 96 Proceedings, pp. 43-54, 1996, and Levoy et al., “Light Field Rendering,” Computer Graphics, SIGGRAPH 96 Proceedings, pp. 31-42, 1996. The best of those methods, as described by Gortler et al., include a visual hull of the object for improved ray interpolation. However, those methods use static illumination, and cannot accurately render objects into novel arbitrary realistic scenes.
An intermediate between purely model-based and purely image-based methods uses view-dependent texture mapping, see Debevec et al., “Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach,” Computer Graphics, SIGGRAPH 96 Proceedings, pp. 11-20, 1996, Debevec et al., “Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping,” Proceedings of the 9th Eurographics Workshop on Rendering, pp. 105-116, 1998, and Pulli et al., “View-Based Rendering: Visualizing Real Objects from Scanned Range and Color Data,” Eurographics Rendering Workshop 1997, pp. 23-34, 1997. They combine simple geometry and sparse texture data to accurately interpolate between images. Those methods are effective despite their approximate 3D shapes, but they have limitations for highly specular surfaces due to the relatively small number of texture maps.
Surface light fields can be viewed as a more general and more efficient representation of view-dependent texture maps, see Nishino et al., “Eigen-Texture Method: Appearance Compression based on 3D Model,” Proc. of Computer Vision and Pattern Recognition, pp. 618-624, 1999, Miller et al., “Lazy Decompression of Surface Light Fields for Precomputed Global Illumination,” Proceedings of the 9th Eurographics Workshop on Rendering, pp. 281-292, 1998, Nishino et al., “Appearance Compression and Synthesis based on 3D Model for Mixed Reality,” Proceedings of IEEE ICCV '99, pp. 38-45, 1999, Grzeszczuk, “Acquisition and Visualization of Surface Light Fields,” Course Notes, SIGGRAPH 2001, 2001, and Wood et al., “Surface Light Fields for 3D Photography,” Computer Graphics, SIGGRAPH 2000 Proceedings, pp. 287-296, 2000. Wood et al. store surface light field data on accurate high-density geometry, whereas Nishino et al. use a coarser triangular mesh for objects with low geometric complexity.
Surface light fields are capable of reproducing important global lighting effects, such as inter-reflections and self-shadowing. Images generated with a surface light field usually show the object under a fixed lighting condition. To overcome this limitation, inverse rendering methods estimate the surface BRDF from images and geometry of the object.
To achieve a compact BRDF representation, most methods fit a parametric reflection model to the image data, see Lensch et al., “Image-Based Reconstruction of Spatially Varying Materials,” Proceedings of the 12th Eurographics Workshop on Rendering, 2001, Sato et al.,” Object Shape and Reflectance Modeling from Observation,” Computer Graphics, SIGGRAPH 97 Proceedings, pp. 379-387, 1997, and Yu et al., “Inverse Global Illumination: Re-covering Reflectance Models of Real Scenes from Photographs,” Computer Graphics, SIGGRAPH 99 Proceedings, pp. 215-224, 1999.
Sato et al. and Yu et al. assume that the specular part of the BRDF is constant over large regions of the object, while the diffuse component varies more rapidly. Lensch et al. fit a Lafortune BRDF model to each point on the object surface. Simple parametric BRDFs, however, are incapable of representing the wide range of reflections seen in real scenes. As described by Hawkins et al., objects featuring glass, fur, hair, cloth, leaves, or feathers are very challenging or impossible to represent this way.
An alternative method uses image-based, non-parametric representations for object reflectance, see Marschner et al., “Image-based BRDF Measurement Including Human Skin,” Proceedings of the 10thEurographics Workshop on Rendering, pp. 139-152, 1999. They use a tabular BRDF representation and measure the reflectance properties of convex objects using a digital camera. Their method is restricted to objects with a uniform BRDF, and they incur problems with geometric errors introduced by 3D range scanners. Image-based relighting can also be applied to real human faces by assuming that the surface reflectance is Lambertian, see Georghiades et al., “Illumination-Based Image Synthesis: Creating Novel Images of Human Faces under Differing Pose and Lighting,” IEEE Workshop on Multi-View Modeling and Analysis of Visual Scenes, pp. 47-54, 1999.
More recent approaches use image databases to re-light models of objects from a fixed viewpoint without acquiring a full BRDF, see Debevec et al., “Acquiring the Reflectance Field of a Human Face,” Computer Graphics, SIGGRAPH 2000 Proceedings, pp. 145-156, 2000, Hawkins et al., “A Photometric Approach to Digitizing Cultural Artifacts,” 2nd International Symposium on Virtual Reality, Archaeology, and Cultural Heritage, 2001, Koudelka et al., “Image-based Modeling and Rendering of Surfaces with Arbitrary BRDFs, “Proc. of Computer Vision and Pattern Recognition, 2001, and Malzbender et al., “Polynomial Texture Maps, Computer Graphics, SIGGRAPH 2001 Proceedings, pp. 519-528, 2001.
They use a light stage with fixed camera positions and a rotating light to acquire the reflectance field of a human face, or of cultural artifacts. The polynomial texture map system described by Malzbender et al. uses a similar technique for objects with approximately planar geometry and diffuse reflectance properties. Koudelka et al. use essentially the same method as Debevec et al. to render objects with arbitrary BRDFs. Those reflectance field approaches are limited to renderings from a single viewpoint. In Debevec et al., a parametric model of skin reflectance is fit to the data to synthesize views from arbitrary directions. As described by Kaudelka et al., that approach does not generalize to complex materials.
In the prior art, a surface reflectance field of an object is defined as the radiant light from a surface under every possible incident field of illumination. It is important to note that, despite the word reflectance, the surface reflectance field captures all possible surface and lighting effects, including surface texture or appearance, refraction, dispersion, subsurface scattering, and non-uniform material variations.
Zongker et al. describe techniques of environment matting to capture mirror-like and transparent objects, and to correctly composite them over arbitrary backgrounds, see Zongker et al., “Environment Matting and Compositing. In Computer Graphics, SIGGRAPH 1999 Proceedings, pp. 205-214, August 1999. Their system is able to determine the direction and spread of the reflected and refracted rays by illuminating a shiny or refractive object with a set of coded light patterns. They parameterize surface reflectance into 2D environment mattes. Extensions to environment matting include a more accurate capturing method and a simplified and less accurate procedure for real time capture of moving objects. However, their system only captures environment mattes for a fixed viewpoint, and they do not reconstruct the 3D shape of the object.
Therefore, it is desired to automatically generate computer graphics models from objects made of arbitrary materials under various lighting conditions, so that the models can be rendered in realistic scenes with arbitrary lighting.